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ABSTRACT 



This review of a research institute's evaluation of 
the national Follow Through program asserts that both the contracting 
office and the research institute failed to adequately specify the 
kind of study intended and the functions which it proposed to serve. 
One of several kinds of evaluation studies might have been 
undertaken: (1) a research study , demonstrating relationships between 

variables to provide hypotheses for program approaches, (2) a 
summative evaluation of the national program, (3) a summative 
evaluation of the relative effectiveness of various sponsor programs 
in achieving the Follow Through objectives, or (4) a formative 
evaluation, providing information to the decision-maker useful in 
modifying the program or in understanding outcome differences. It is 
difficult to make specific recommendations because of the lack of 
study definition. General recommendations are: (a) that the 

institute's evaluation study be continued for one more year since 
preliminary data collection has already been done , (b) the study 
focus be clarified, with the assistance of an outside panel, and (c) 
the best approach for the coming year would probably be alternative 
(3) , listed above. (Author/NH) 
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The Follow Through program was established under the Economic 

Opportunity Act of 1964 and subsequent amendments. The act authorized: 

A program to be known as Follow Through focused prima- 
rily on children in kindergarten or elementary school 
who are previously enrolled in Head Start or similar 
programs and designed to provide comprehensive services 
and parent participation activities .. .which the direc- 
tor finds will aid in the continued development of 
children in their full potential. . . [Section 222(A)] 

There are currently 161 Follow Through projects in 140 communities. 
Nineteen persons or organizations function as Follow Through program 
sponsors, i.e., as the designers or '•model makers" of the experimental 
approaches that make up Follow Through. Those who have devised the 
program maintain that each Follow Through project is an example of delib- 
erate attests to field a wide array of experimental compensatory programs . 
In suranary, the Follow Through program nationally is composed of what is 
purported to be "161 experiments in education theory and practice placed 
in 140 communities which adapt them according to the local socio- cultural- 
political context." 

The United States Office o c Education contracted with a research 
institute "to develop information useful in judging the extent to which 
each project is bringing about desirable changes in students and the 
communities." The institute has been engaged in this evaluation project 
for two years. The intent of this paper is to provide a review and 
critique of the work performed by the institute . 



The greatest deficiency of the evaluation project is its failure 
to provide adequate specification of the kind of study it is intended 
to be and the functions which it proposes to serve. The evaluation 
group cannot be held alone responsible for this deficiency. The Follow 
Through project is itself a research and development activity and, as 
such, has undergone considerable modifications during the past years. 

As a consequence, partially, of this lack of definitiveness in program 
description and, because of other general changes that might have occurred 
in the project, the responsible Office of Education officials have not 
provided adequate specification of the objectives and purposes of the 
study. The research institute, however, is not blameless; if they accepted 
four million dollars in evaluation funds, it should be expected that they 
would have been more aggressive in demanding clarification of the objectives 
and the decision-making purposes. 

There appears to be faulty conception on the part of the institute, 
as well as Follow Through officials, regarding the various purposes served 
by evaluation studies in contradistinction to research studies, and in the 
kinds of evaluation studies that might have been undertaken. There are 
several courses that the study might have followed: 

1. Research Study . It might have been felt that the nature of the 
program was relatively exploratory, that there were few clear cut objectives 
relative to the anticipated outcomes of specific Follow Through programs, 
and that what was most important was an analysis of the relationships between 
inputs to systems and between various pro~e~,ses (procedures) employed within 
programs, and a wide variety of potentially important outcome dimensions. 



Such a study would not have required the previous selection of outcomes 
nor the specification of specific decision areas of concern on the part 
of a decision-maker or body. Rather, it would have been viewed as a re- 
search study which would demonstrate interesting and informative rela- 
tionships between variables to provide hypotheses that might be 
considered in developing a specific number of finite program approaches 
considered potentially most feasible. In light of the failure of the 
institute to state adequately the objectives under examination and, 
instead, to concentrate on a wide variety of variable dimensions, one 
must conclude that the above function, perhaps, is what they most 
nearly had in mind. 

2. Summative Evaluation: The National Program . On the other hand, 

if one is to take seriously the title of the proposal, "Longitudinal 
Evaluation of the National Follow Through, 1970-71," then one must be 
dismayed at what is proposed; for such a title suggests that what is to 
be evaluated is the national program. If that is the case, then the implicit 
policy question relates to the extent to which the program (namely, the 
total national Follow Through program) is achieving the objectives stipulated 
in the legislation. Such evaluative information, one might conclude, is 
necessary in order to reach decisions, for example, about whether Follow 
Through should be expanded, modified, maintained, or curtailed. If this 
were the desired function of the study, then the procedure would be to 
obtain from the Follow Through program directors the operationally stated 
objectives of the national Follow Through program which they had developed 
as "the first step" in operationalizing the intent of Congress as implied 
from the legislation. If these objectives had not been developed, then 



it would, appear that the institute's first responsibility would be to 
develop such operational objectives in concert with those in the Follow 
Through program. The main purpose of the study, in such an instance, would 
be to determine the extent to which the national program, in all its com- 
plexity and with all its variations, is achieving the stipulated objectives 
of the program. If this approach represents the intended evaluation 
function, then one of the main criticisms must center on the inappropriateness 
of the evaluator reporting directly to the Follow Through branch. It does 
not appear to be a particularly viable course of action to provide a situa- 
tion where the evaluator reports to the individuals whose work he is critiqu- 
ing. It would have been more appropriate to have the institute report 
directly to EESE or to the PPE section of EESE. 

If this national evaluation is indeed the intent, then the second 
main difficulty of the proposal is the failure to state the objectives of 
the national program. Without such a list of objectives, operationally 
stated, one can only guess as to the relevance of the outcome variables and 
the measures selected by the institute and the reason for their use. 

3. Summative Evaluation: The Sponsor Programs of National Follow 
Through . A third major function that might be served by the study would 
be the evaluation of the relative effectiveness of various sponsor programs 
in achieving the Follow Through objectives. That is, one considers that 
the Follow Through branch has a number of different program approaches, 
each designed for accomplishing the objectives of the national Follow Through 
program. (For, if they were not intended to accomplish the objectives of 
the national program, surely they would not have been accepted as alternatives) . 



PS 003668 



t 



- 5 - 

In this instance, the kind of decision envisioned by the decision- 
maker as a result of the evaluation information is related to the potential 
deletion of programs, encouragement of others, or perhaps mandating of 
a particular approach . If a summative evaluation of the type specified 
in this section is indeed what was desired, then there are, again, several 
modifications of the present proposal that would be in order. For example: 

a. The title of the study might well be changed to M An Evaluation 
of the Sponsor Programs in the National Follow Through Program." 

b. There is a need to state in operational terns the objectives 
of the national Follow Through program. 

c. A procedure needs to be determined for ordering the objectives 
in teims of priority. This is necessary because of the likeli- 
hood that the list of specified objectives would be so great 
that it would not be physically possible to measure all of them. 

d. If limitation of the objectives is necessary, the research 
institute shculd select the objectives of highest priority for 
further examination. 

e. The appropriate measures for each of the selected objectives 
should be determined. (While the institute has selected a number 
of variables descriptive of outcome dimensions, it is not possible 
to determine whether these measures are appropriate without 
having first determined the objectives). 

f . Modifications in the collection and analysis of information would 
undoubtedly need to be made, in terms of the study previously 
alluded to, in order to provide the desired kind of relevant 
information to the decision-maker. 
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4. Formative Evaluation . The second and third studies previously 
specified are summative kinds of evaluation studies. That is, at the con- 
clusion of the program, they provide information to the decision-maker about 
the performance of the program. Among those working in the area of evaluation, 
a distinction has been made between summative and foimative kinds of evalua- 
tion. The purpose of a foimative study is to provide infoimation to the 
decision-maker, during the course of the operation of the program, in order 
to allow for modifications of the program at that time. The Center for the 
Study of Evaluation, UCLA, has referred to foimative kinds of evaluation 
as "implementation evaluation" and "program improvement evaluation." These 
teims emphasize the responsibility of the evaluator for providing infoimation 
of two types. In the first instance, the evaluator is called upon to provide 
information about the extent to which the program has been implemented in 
the manner in which it was described and to the group for whom it was intended. 
That is, are the various sponsor programs being implemented in the manner 
in which they were described by the sponsors, and do the groups participating 
in these programs possess the characteristics intended for the particular 
program? For example, if an Engelmann program has characteristics A, B, C, 

D, E, F, and G and is intended for a student population having characteristics 
S, T, U, V, and W, are the characteristics of the program present when imple- 
mented , and are the characteristics of the students participating in the 
program roughly approximate to those for whom the program was intended? 

Knowledge of this kind of infoimation gathered from an implementation eval- 
uation is of value to the decision soaker in modifying the program or, at 
a later time, in understanding outcome differences in programs. The institute's 



study has collected sane data of this type; however, it has been viewed 
as data which is part of a variable set to be used in explaining outcomes 
(approach number 1) rather than as information to be provided to decision- 
makers during the course of the program. Improvement, or process evaluation, 
provides information to decision-makers about short term outcomes or progress 
in the program, in order that modifications might be made at that time, 
rather than allowing a glaring deficieacv to exist for a full year. 

One might conceive of implementation and improvement evaluations related 
either to the evaluation of the national Follow Through program (alternative 2) 
or implementation and improvement evaluations related to the evaluation of 
the program^ Follow Through (approach 3) . 

Recommendations 

In light of the lack of definition of what kind of study is being requested 
of the research institute, it is difficult to make specific recommendations. 
However, I will attempt to make some recommendations, general in nature, and 
to specify the kind of study intended when specific recommendations are made: 

1. There are several kinds of evaluations that might be appropriate, 
and which are very much needed considering the expenditure of funds for the 
Follow Through program. In my view, the work done by the institute to date 
is primarily related to approach number 1 (above). It does not serve, there- 
fore, the purposes of evaluation specified in approaches 2 or 3. Thus I cannot 
vouch for the worth of the approximately four million dollars already expended. 
However, the experience that the institute has gained to date in understanding 
the program, in working with Follow Through sponsors, and with Local Educational 



Agencies places them in a superior position to conduct a reasonable eval- 
uation within the next year, well worth the dollar expenditure anticipated 
for that year. Moreover, they have already gathered data which would, 
presumably, be useful in a future evaluation. Thus, the first general 
recommendation is that the institute's Evaluation Study of the Follow Through 
Program should be continued for one more year . 

2. Prior to the collection of any further data, prior to the development 
of additional instruments, prior to additional arrangements for field testing, 
prior to any further analyses of data, there is a necessity for the focus of 
the study to be clarified . I do not believe that this can be done by the 
parties involved without outside assistance. I would, therefore, suggest 
that either a panel be convened to work with the institute and Follow Through 
officials in mapping the evaluation plan for next year, a contract be extended 
for such a purpose, or that the PPE Office of EESE obtain advice and assist 

in the reformulation of the project. 

3. Without attempting to second guess the nature of the refoimulation 

of the project, it would be my view that a Summative Evaluation: The Sponsor 

Programs of National Follow Through (alternative 3, page 4) would be the 
most appropriate approach for the coming year. If this is the case, then 
the modifications of the present proposal suggested within the comments on 
that section perhaps could be further clarified and developed. 



